LUCENE-14611: Reduce memory pressure in TestTermInSetQuery.testDuel#16300
Conversation
|
Hi @dweiss, thanks for filing this issue. You reviewed the earlier attempt in #15118, so I thought you might want to take a look at this one. It takes a different approach from the range-capping in #15118. Instead of shrinking the randomized inputs, it goes after the memory pressure itself. Two changes:
The original stress shape stays the same. I reproduced the OOM under a 16MB heap and it passes after the change. Let me know what you think when you get a chance. |
|
This looks great, thank you. |
| assertEquals(td1.scoreDocs[i].doc, td2.scoreDocs[i].doc); | ||
| if (scores) { | ||
| if (scores) { | ||
| final TopDocs td1 = searcher.search(q1, maxDoc); |
There was a problem hiding this comment.
For pure topDocs use it would be enough to use numDocs, because deleted documents - we don't care!
Of course this does not matter if the index has no deletions (not sure about this test). I just noticed this.
There was a problem hiding this comment.
Thanks, that's a good point.
|
I've backported all changes to this test to 10x. |
Description
This reduces avoidable memory pressure in
TestTermInSetQuery.testDuel.The test is checking query equivalence, not query-cache behavior. It still uses the randomized test-framework
newSearcher(reader), but disables query caching for this test so repeated equivalence checks do not retain cached doc-id sets across iterations.For the no-score comparison path, the test now collects matching doc IDs with the existing test-framework
FixedBitSetCollectorand compares the resulting bit sets. This avoids materializing full sortedTopDocswhen only doc-id equality matters.This is intentionally different from prior range-cap attempts such as #15084 and #15118: it does not reduce randomized input sizes or change the stress shape. Instead, it removes the pressure sources that are outside the behavior this test is meant to validate.
Local investigation
I could not reproduce the OOM on my machine with the original issue command, likely due to local environment differences. To make the failure mode visible locally, I used the same stress shape with the test JVM constrained to a 16m heap.
With the original code, that 16m squeeze reproduced:
Heap dump inspection,
jstat, and JFR heap profiling pointed to avoidable pressure from the randomized query cache retaining doc-id sets and from the no-score branch materializing sortedTopDocs.Verification
Ran with JDK 26.
I also re-ran the local 16m squeeze with the inner
testDuelloop temporarily raised from100to10000; with this fix applied, it completed successfully instead of reaching the Java heap OOM state observed with the original code.Fixes #14611.